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Summary and Recommendations 


Purpose and Strategy 

Extending the school day through programs that will supplement 
core courses, such as offering intensive, after-school tutoring or 
small-group study sessions in reading and mathematics, is one 
method districts use to boost school performance and narrow 
educational achievement gaps between students of different racial 
and economic backgrounds (Redd et. al, 2012). In 2010-11, the 
Wake County Public School System (WCPSS) K-12 Intervention 
Services launched the Academic Achievement Academy (AAA). 
The purpose of AAA was to raise student academic growth and 
proficiency levels at elementary and middle schools with low 
ABCs performance composites. It was expected that this rise in 
student achievement would help schools meet the overall 
program goal of increasing their performance composites to at 
least 70%. The performance composite reflects the percentage of 
End-of-Grade (EOG) and End-of-Course (EOC) exam scores in 
the school that are considered at or above grade level or 
proficient, which equates to scale scores that fall into 
achievement Levels III and IV. 


The Academic Achievement Academy 
(AAA) was initiated in 2010-1 1 . It 
provided short-term, after-school tutoring 
in reading and mathematics for students in 
grades 3-8 with the goal of increasing the 
performance composites of participating 
schools to at least 70%. In 201 1-12, AAA 
supported 1,185 students at 20 schools, 13 
of which also participated in AAA in 
2010-11. By the end of 201 1-12, four 
first-year and four second-year schools 
had met the program goal. Findings 
indicate less of a positive impact on 
reading outcomes than mathematics, 
based on comparisons to a matched group 
of students. AAA cost about $500 per 
student served in 201 1-12, increasing to 
$864 for each student who made 
academic growth. AAA was discontinued 
for the 2012-13 school year. 
Recommendations for future after-school 
program implementation efforts are 
offered. 


The strategy AAA implemented to achieve this goal was the 
provision of short-term, after-school tutoring in reading and 
mathematics to students who needed additional support. The 
academy specifically targeted students in grades 3-8 with Level II 
and low Level III achievement levels on the previous years’ 
reading or mathematics EOG exams, and students who were not 
currently meeting grade-level reading or mathematics benchmark 
standards and were at-risk of not meeting promotion standards. 
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Major Findings 

AAA had a second year of implementation in 201 1-12. The AAA spring 2012 academy 
provided after-school instructional resources to 20 elementary and middle schools with low 
ABCs performance composites. Overall, there were 1,185 students at those schools who 
received reading or mathematics tutoring from AAA, which was very close to the K-12 
Intervention Staff’s goal of serving 1,200 to 1,300 students. Among the students with prior EOG 
data, most (77.3%) met the program participation criteria of performing at an EOG relative 
achievement level between 2.0 and 3.3. Students performing within these achievement level 
ranges were expected to receive the most benefit from the short-term, targeted tutoring. 

Did AAA help to raise student achievement at the targeted schools? 

Analyses at the student level indicate that AAA tutoring had almost no impact on EOG 
proficiency and growth in reading and no more than a small, positive influence on mathematics. 

It appears that AAA was most effective among Level III students, which is where most of the 
small gains were found. It is not uncommon for programs to be more effective in yielding 
positive outcomes in mathematics than in reading. In their meta-analysis of summer programs 
that were similar in length and intensity to many after-school programs, Cooper et al. (2000) 
found less favorable outcomes on reading assessments than mathematics 

Within this evaluation, reading outcomes typically showed either slightly greater results for the 
students who did not receive AAA tutoring or no difference between groups. 

• Slight gains in proficiency between 2010-11 and 2011-12 were most commonly due to 
movement from Level II to Level III among both groups, with greater increases found for 
comparison students. The increases in the percentage of Level III students were primarily at 
high Level III (3.3 to 3.92) among the AAA group and at low Level III (3.0 to 3.29) among 
the comparison group. 

• There was no overall difference in the percentage of AAA and comparison students meeting 
growth targets in 201 1-12. 

• Based on logistic regression results, being proficient on the 2011-12 reading EOG exam 
significantly increased the odds of meeting growth in reading that same year. The extent of 
this impact was the same regardless of whether students participated in AAA or not. AAA 
reading tutoring did appear to have a minor, yet statistically significant, positive impact on 
LEP students making growth. 

AAA participants had slightly better mathematics outcomes, in terms of reaching proficiency and 
meeting growth targets, than their matched counterparts. 

• The one-year gains in proficiency were primarily due to movement from Level II to Level III 
among both groups. Again, for AAA students these increases are seen at high Level III (3.3 
to 3.92) and at low Level III (3.0 to 3.29) for comparison students. 
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• There was a statistical difference in the percentage of Level III students who met their 
mathematics growth targets favoring the AAA group. It appears that the performance of the 
high Level III students contributes to most of this positive change. 

• The odds of making growth in mathematics in 201 1-12 were higher for students who 
received mathematics tutoring from AAA and were proficient on the EOG exam in the spring 
of 2012 than for proficient students who did not participate in AAA. 

Table 1 

Comparative Summary of Student Achievement Outcomes by Subject 


Outcome 

Reading 

Mathematics 

Gains in proficiency before and after 
AAA participation 

Slightly lower for AAA group 
than comparison group 

Slightly greater for AAA group 
than comparison group 

Meeting growth targets in 2011-12 

No difference between 
groups 

Slightly greater for AAA group 
than comparison group 


Were AAA schools able to increase their performance composites to at least 70 %? 

A key goal of AAA was to help the district’s lowest performing elementary and middle schools 
raise their performance composites to at least 70%. 201 1-12 was the second year of AAA 
implementation for 13 of the 20 participating schools because they did not raise their 
performance composites to at least 70% during the first implementation year (2010-11). Based 
on end-of-year data for 201 1-12, eight of the 20 participating schools (40%) met the program 
goal. Half of these schools were first-year AAA schools and the other half were second-year 
AAA schools. Statistically significant gains in the percentages of students who scored proficient 
after receiving AAA tutoring were found at very few schools. Regardless of the statistical 
impact, the schools with the highest percentages of AAA students meeting growth, in most cases, 
also had the greatest improvement in AAA student proficiency. 

Did the tutoring dosage based on students’ attendance at the AAA tutoring sessions have 
any impact on their reading or mathematics outcomes? 

Current research has presented evidence of the importance of high student attendance at 
remediation intervention programs to reap the greatest benefits. Redd et. al (2012) reviewed 
experimental, quasi-experimental, and non-experimental studies on the impact of extended 
school-day programs, including targeted interventions for low-performing students and 
additional hours of core classes or intensive tutoring after the school day, on academic 
achievement outcomes such as standardized test scores. They present findings which suggest 
that these programs are more effective if student participation rates are high. Results from 
studies on summer school programs that are similar in duration and intensity to after-school 
programs can also be informative. Based on their literature review on summer learning loss and 
the effectiveness of summer learning programs, McCombs et al. (2011) cited various studies that 
found a positive relationship between attendance and outcomes. 
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Overall, most AAA students (82%) participated in the program at least 75% of the time, which 
amounts to attending a minimum of 15 sessions lasting 90 minutes each (totaling 22.5 hours). 

A greater attendance rate was found among elementary students than middle school students. 
Based on survey results, 67% of site coordinators responded that students were rarely absent 
from tutoring sessions. 

The findings presented in this report indicate that students’ level of attendance (dosage) in the 
AAA tutoring intervention did not positively impact their EOG proficiency or academic growth 
for 201 1-12. Students with high attendance did not have more positive academic outcomes than 
those with low attendance. 

To what extent did students from the AAA spring 2011 academy cohort reach proficiency 
by the end of the 2011-12 school year? 

Most of the students who received support from AAA in 2010-11 were not proficient a year 
later. For instance, a third of the students who received reading tutoring were proficient in 201 1- 
12. Among the students tutored in mathematics, 49.5% were proficient, representing a one-year 
statistically significant decline of about 10 percentage points between 2010-11 and 2011-12. 

In general, the proficiency rates appear to be greatest during AAA implementation (2010-11) and 
lowest the years before and after. Research has suggested that the effects of remediation 
interventions may be most lasting, if “booster” sessions are available (Snow, Bums & Griffin, 
1998). It is possible that AAA contributed to this slight bump in proficiency and that students 
might have benefited from some continued support in 201 1-12, especially for reading. For 
instance, a small percentage of AAA students who still had low levels of proficiency after 
receiving reading tutoring in 2010-11 participated in AAA reading intervention in 201 1-12. 

This group’s reading proficiency doubled after a second dose of tutoring. 

What are the site coordinators’ perspectives of AAA implementation in 2011-12? 

The site coordinators at participating schools provided positive feedback on the implementation 
efforts of AAA. They reported high levels of student and tutor engagement, district and school 
level support, and a general satisfaction with the program. The materials were viewed as both an 
effective program element and as an area of improvement. 

What was the cost of AAA per student served? 

The cost of the program implementation overall was about $500 per student served. The cost of 
providing tutoring through AAA during the spring 2012 academy increases considerably if the 
cost is calculated per student who made academic growth ($864) or per proficient student 
($1,152). The provision of reading tutoring is more expensive than mathematics tutoring when 
considering the success of the participant. 
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Study Limitations 

There are a few study limitations which may have influenced the findings presented in this 
report. 

Within this report, the examination of changes in student proficiency rates after AAA 
implementation is restricted to students who have prior EOG results, thus excluding grade 3. 

As such, the total impact of AAA on all participants is not presented. Grade 2 data from The 
Dynamic Indicators of Basic Early Literacy Skills (DIBELS) was not used to assess academic 
performance prior to AAA participation because this set of measures for assessing the 
acquisition of early literacy skills were piloted in WCPSS in 2009-10 and used inconsistently in 
2010-11. Moving forward, consistent district implementation of mCLASS:DIBELS Next for 
K-3 literacy and possibly K-3 mathematics provides the opportunity to use these data as pre- 
intervention measures. 

This evaluation compares the outcomes of AAA students to those students who did not receive 
AAA tutoring, yet were demo graphically similar to the participants. Although the matched 
sample is similar to the AAA group, it could have been more tightly aligned if different sample 
selection methods had been employed. For instance, EOG achievement levels were used to 
obtain a sample of students at equivalent levels of proficiency as the AAA participants. Whereas 
the percentages were identical for AAA students and their matched counterparts at achievement 
Level III, a larger percentage of AAA students fell within the low Level III range, which was 
part of the service criteria. This difference may have influenced the findings drawn between the 
groups. A better method might have been to match students by EOG relative achievement level 
or scale scores as closely as possible. Since the AAA program was intended to target students 
within specific achievement levels, drawing a comparison sample from a group of students 
falling within achievement Level II and low Level III first, and then matching by demographic 
characteristics may have yielded a comparison group that more closely represented the 
population AAA was expected to serve rather than the one it did serve. 

A prevalent standard for educational research is to examine whether one program is more 
effective than another similar program (Schneider et al., 2007). As the paragraph above 
explains, the analyses used in this evaluation of AAA draws comparisons between the 
intervention participants and a matched group of students who did not receive AAA tutoring, yet 
may or may not have received other reading or mathematics interventions. Comparisons are not 
made between AAA and other program models. The Data and Accountability (D&A) 
Department is planning to conduct a comparative analysis examining the effectiveness of several 
after-school programs in WCPSS. This will allow various stakeholders to better assess the value 
of one program over another. 

Recommendations 

AAA was not continued in the 2012-13 school year; nevertheless, the evaluation findings from 
its implementation in 2011-12 can inform policy and implementation efforts of future after- 
school instructional programs within WCPSS. The following recommendations are offered to 
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help improve such efforts. Additionally, if AAA were to be implemented again, D&A suggests 
that modifications be made to the program based on these recommendations. 

Offer after-school remediation that is sufficient in duration and intensity. According to 
research, the duration and intensity of after-school programs matter. It appears that short-term 
interventions, particularly for reading, are not effective, especially in the longer-term. 

In their meta-analysis of 53 studies measuring student achievement in reading and mathematics 
after-school programs, Lauer et al. (2004) found that programs that were implemented for at least 
45 hours had statistically significant positive results. Such programs should not run indefinitely 
though. The results started to decline as reading programs exceeded 210 hours and mathematics 
program ran for more than 100 hours. AAA provided 30 hours of intervention support in either 
reading or mathematics, which appears to be insufficient in length to make a sizable impact on 
student achievement alone. 

Establish realistic short-term, intermediate, and long-term goals for program outcomes. It 

is possible that WCPSS after-school tutoring programs such as AAA are helpful to the students 
who participate in them, even if there are limited statistically significant, positive impacts on 
state end-of-year test scores for proficiency and growth. In their review of after-school 
programs, Hammond & Reimer (2006) indicate that positive impacts should not be expected for 
at least six months after program implementation. They also indicate that improved test scores 
should not be the only or even the primary program goal since these programs involve much less 
time and intensity than classroom instruction. Typically, the after-school program impact on 
achievement exams is small. Focusing on intermediate outcomes such as improvements in 
homework completion, grades, higher-order thinking, content knowledge, and study habits 
appear to be more appropriate (Granger & Kane, 2004). 

Use quality materials aligned with the North Carolina curriculum. After-school programs 
may be more effective at producing reading and mathematics outcomes if their instructional 
materials encompass a well-defined reading and mathematics curriculum that is aligned with 
North Carolina and the WCPSS Curriculum Management Application (C-MAPP). Other after- 
school program delivery models might be considered as well. WCPSS could provide extra 
support for homework within the core curriculum, rather than relying on newly structured 
programs based on additional instructional materials. 

When additional materials are used, they should be flexible to both teachers for differentiation 
purposes and students to promote engaged participation. Although the materials used for AAA 
implementation in 2010-11 were found to be highly engaging with some evidence of positive 
impact in different contexts than AAA, K-12 Instructional Service staff selected different 
materials for the 2011-12 implementation that provided opportunities for greater instructional 
differentiation for students. Ideally, the materials chosen may also include instructional 
assessments that can be administered immediately prior to, during, and after the intervention to 
help gauge improvements or changes in student performance. During its first year of 
implementation (2010-11), the materials used to implement AAA included a pre- and post- 
intervention assessment to measure short-term student progress in reading and mathematics. In 
general, somewhat positive trends were found between assessments. There were no coordinating 
assessments for the materials used in 201 1-12. 
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Background 


In 2010-11 and 201 1-12, AAA operated for 1 1 weeks, beginning in mid February and running 
through the end of April/beginning of May, with 10 weeks devoted to instruction. Students 
attended either reading or mathematics tutoring for two days each week. Each tutoring session 
lasted 90 minutes. Students who had perfect attendance at the academy would receive 30 hours 
of reading or mathematics instruction in addition to what they received in the classroom. These 
additional resources were expected to give students the extra support needed to bolster their 
performance on the EOG exams, which were administered a few weeks after the final academy 
session. 

The 2010-11 academy supported over 1,300 students in grades 3-8 from 24 schools that had a 
performance composite of less than 70% in 2009-10. An evaluation conducted by D&A (Baenen 
& Lougee, 2011) found that at the end of the 2010-11 school year, 9 of the 24 schools had 
performance composites that exceeded 70%. Results also indicated that when compared to their 
performance just prior to the academy, a higher percentage of fifth through eighth grade 
participants reached their reading and mathematics growth targets after their participation in 
AAA. No comparisons were drawn between AAA participants and similar students who did not 
receive the intervention. Although this method was considered, it was not used because the 
program was in its first year of implementation and there were also concerns about the 
availability of “complete information on additional support services” the control group might 
have received (Baenen & Lougee, 2011). The progress of AAA students were compared to the 
district overall and the materials used for implementations included a pre-post assessment that 
allowed short-term student progress to be monitored. 

The recommendations emphasized in the 2010-11 evaluation were considered by K-12 
Interventions Services staff. Specific changes to the second year of implementation in 201 1-12 
were made based on the following recommendations: 

1. Select and place students more strategically. 

2. Enhance recruitment of value-added teachers and refine selection criteria for other highly 
qualified teachers. 

3. Provide a tighter match of instruction to student needs through grouping strategies and 
greater differentiation. 

In response to these recommendations, K-12 Interventions Services staff utilized student and 
teacher data provided by D&A to assist principals and site coordinators in identifying students 
eligible for participation and highly effective teachers to recruit as tutors. The staff also 
encouraged a more systematic selection process of assigning participating students to tutoring 
groups. For instance, tutoring groups were created for each subject based on student ability or 
need rather than grade level and each tutor was assigned a group with no more than 6 students. 
These modifications are discussed in greater detail in the following section of this report. 
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The K-12 Interventions Services staff also made the following changes in preparation for the 

second year academy. 

• Principals of 22 elementary and middle schools were selected to participate in AAA during 
the spring of 2012 based on the school’s performance composite for 2010-11. Unlike the 
first year academy, these school principals were given the option of accepting or declining 
the invitation to secure better buy-in of the program and because they may have had other 
intervention programs already in place. 

• A teacher to student ratio of 1 to 6 (rather than 1 to 7) was implemented to promote more 
time for individual attention and to allow tutors to pair off students to work collaboratively. 

• Schools projected to have at least 100 participating students had the option of hiring two site 
coordinators who could share the responsibilities of program implementation and oversight. 

• To provide high quality materials that also gave tutors the flexibility to differentiate 
instruction according to the needs of their students, K-12 Intervention Service staff partnered 
with Academics Department staff to select new research-based materials for delivering 
reading and mathematics remediation in the spring 2012 academy. 


Spring 2012 Academy 

School Selection 

Invitations to participate in the spring 2012 academy were sent to the principals of 21 elementary 
and middle schools with 2010-11 performance composites below 70%. The K-12 Intervention 
Services staff also extended an invitation to Walnut Creek Elementary School. The staff thought 
AAA would be an appropriate source of support during Walnut Creek’s opening year in 201 1- 
12, given its projected enrollment of a large proportion of students with academic risk factors. 

Two of the principals whose schools participated in the spring 2011 academy declined a second 
year of participation, resulting in 20 participating schools for the spring 2012 academy. Among 
these 20 schools, 6 elementary schools and River Oaks Alternative Middle School were first year 
participants. The other 13 schools participated in AAA during the 2010-11 school year and were 
invited to hold academies for a second year because they had not yet reached a performance 
composite of 70%. Although the majority of these 13 schools had increases in the percentage of 
proficient exam scores between spring 2010 and spring 2011, these gains were not sufficient 
enough to raise their performance composites to 70%. 

Staff Selection 

Principals at each of the 20 participating schools selected a staff member to be their AAA site 
coordinator in charge of program implementation and oversight. Schools that were allotted 100 
or more participant slots were allowed two site coordinators. All site coordinators received the 
same compensation. Site coordinators typically held school-based positions such as classroom 
teachers and instructional resources teachers (IRTs). 
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Teachers from across the district were hired as academy tutors to deliver reading or mathematics 
remediation to program participants. The number of tutors at each site was allocated depending 
on the number of students served and was expected to yield a 1 to 6 teacher to student ratio. 
Within the recruitment process for the 2010-11 academy, teachers who were identified as highly 
effective based on ratings from the Education Value-Added Assessment System (EVAAS) were 
given priority. Due to confidentially and privacy issues surrounding teacher level EVAAS data, 
these data were not available to be used for teacher recruitment for the 2011-12 academy. 
Instead, D&A provided K-12 Intervention Services staff with teacher- level data to assist in tutor 
recruitment. These data showed the percentage of students who had met their growth targets in 
reading and mathematics for the 2010-11 school year by the classroom teacher for grades 4 
through 8. Teachers who served at least 12 students and who had at least 70% of them making 
growth in reading or mathematics were considered candidates. 

Selected teachers were invited to attend an AAA information session held in mid January, at 
which time they could sign-up to participate as a tutor for the spring 2012 academy. Although a 
large percentage of the recruited candidates attended the information session, many did not 
accept the tutor position. Recruitment of tutors for the spring 2012 academy may have been 
impacted by two changes in the way they would be compensated. In 201 1-12, there was a flat 
$35 hourly pay rate for all teachers hired as tutors and with no possibility of being reimbursed 
for any travel expenses incurred. These compensation conditions differed from those available to 
AAA tutors for the 2010-11 school year. For example, spring 201 1 academy teachers who were 
identified as highly effective (based on ratings from EVAAS) received a higher hourly pay rate 
than other teachers who were hired as tutors, and teachers who were hired to tutor at a different 
school received travel reimbursements. 

Since the first recruitment method did not secure an adequate number of tutors, a second 
information session and alternative recruitment techniques were used. There were several ways 
that the K-12 Intervention Services attempted to fill the gap in tutoring positions: principals 
were asked to recommend teachers, current high school teachers and retired teachers who 
previously worked in the district were solicited, and some hired tutors were asked to work at two 
academy sites on alternating days of the week. All tutors who were eventually hired attended an 
AAA training session held on February 4, 2012. A site-based meeting with site coordinators and 
their tutors was also held prior to the first tutoring session. 

Student Selection 

One goal of AAA was to serve between 1,200 and 1,300 students in the spring 2012 academy. 
Each of the schools was allotted a certain number of students, based on the needs of each school, 
who could attend the academy. D&A provided data to each school indicating the number of 
proficient EOG and/or EOC exam scores needed to reach a performance composite of 70% in 
201 1-12. These data helped to inform allotment decisions. Principals were informed of the 
number of students who could be served in the spring 2012 academy. 

In an effort to establish a systematic process for selecting program participants, D&A staff also 
provided student-level academic performance data to each school. Each school’s roster 
identified current students who earned a scale score within achievement Level II or a low Level 
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III on the reading EOG or the mathematics EOG in 2010-1 1. A low Level III was 
operationalized as a relative achievement level between 3.0 and 3.3. Relative achievement levels 
are a metric that indicates where a student's scale score falls within each achievement level by 
attaching decimal places to the usual achievement level designation. For example, a relative 
achievement level score of 3.48 would indicate that a student's scale score is in the middle of the 
Level III range for that test. As such, 3.3 was designated as the upper limit of a low Level III, 
the highest achievement level for which students were eligible for AAA participation. A total of 
863 students comprised the rosters. 

Principals and site coordinators were instructed by K-12 Intervention Services staff to use the 
student rosters to select program participants, with certain caveats. The vast majority of 
elementary and middle school students listed on the rosters were in grades 4 through 8 in 201 1- 
12. Some third grade students who fell within achievement Level II and were retained in grade 
were also identified. Staff also selected other students who were not identified on the rosters but 
were eligible for participation, such as third grade students who were not meeting grade level 
reading or mathematics benchmarks and were at risk of being retained and transfer students who 
were academically at-risk. Principals were also asked to consider the intervention services 
students were already receiving when selecting AAA candidates and to avoid pulling students 
from those programs, especially if the programs were also after school with a focus on reading 
and mathematics remediation. Inspection of the participant list indicated that staff did use the 
provided rosters to select most of the student participants. 

Once program candidates were selected, an information letter was sent inviting parents to enroll 
their children in the academy. Program incentives such as offering a low teacher to student ratio 
(1 to 6) and providing an after-school snack and transportation home from the academy were 
specifically mentioned to encourage participation. A waiting list for each academy site was 
created if all allotted slots were filled and there was still an interest in enrollment. 

Tutoring Schedule 

Principals at each participating school choose between a Monday/Wednesday and a 
Tuesday/Thursday schedule for the 11 week academy. Scheduling adjustments were made to 
account for missed sessions due to teacher workdays and track-out days for year round students. 
As such, students might attend 3 or 4 days of after-school tutoring during pre-specified weeks. 
Ten of the 11 weeks were specifically reserved for instruction, equaling 20 tutoring sessions. 
Scheduling for the final week included one day for review and one day for celebration, resulting 
in a maximum of 22 total sessions. School principals also determined whether their site would 
offer tutoring in reading, mathematics, or both subjects depending on the remediation needs of 
the targeted students and the type of programs currently offered at the school. Individual 
students would, however, receive tutoring in one subject only. 

Instructional Materials 

Based on the evaluation results of the spring 201 1 academy, the Voyager materials used to 
implement AAA in 2010-1 1 were not chosen for the 2011-12 school year because they did not 
sufficiently allow for differentiation of instruction nor were they judged to be the best resources 
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to use for brief programs, such as the 1 1 week academy. The instructional materials used for 
AAA in 201 1-12 were recommended by WCPSS Academics Department staff. The materials for 
reading were Reading for Information at grades 3-5 and Read Excel at grades 6-8. Students in 
grades 3-8 received mathematics tutoring based on materials from a company called Teacher 
Created Materials. Although the materials were deemed appropriate for providing reading and 
mathematics remediation to the targeted students, there was no formal evidence of their 
alignment to the North Carolina EOG assessments. 
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Methodology 


A central purpose of this report is to examine whether there was an increase in the proportion of 
AAA students who met their growth targets and proficiency benchmarks at the end of the 2011- 
12 school year, which in turn was expected to raise the participating schools’ performance 
composites to at least 70%. Several research questions guide the analyses of AAA 
implementation and academic outcomes, as shown in Table 2. Additional analyses were also 
conducted to address several research questions that arose during the evaluation process. Basic 
analyses calculating the cost of providing AAA tutoring to students was also conducted. The 
research questions are investigated using a variety of data sources and methods which are 
described below or in subsequent sections of this report. 

Table 2 

Research Questions 


Area of Analysis 

Research Question 

Implementation 

What is the enrollment and attendance data of the students served? 
What are the characteristics of the students served? 

What are the site coordinators' perspectives of AAA implementation in 
2011-12? 

Academic 

Outcomes 

Did the participating students improve their EOG reading and 
mathematics proficiency levels in 2011-12? 

Did the participating students reach their End-of-Grade (EOG growth 
targets for 2011-12? 

Did AAA students achieve stronger EOG outcomes than students with 
similar characteristics who did not receive the service? 

Did the schools participating in the spring 2012 academy raise their 
performance composites to at least 70% by the end of the 2011-12 
school year? 

Additional Analyses 
of Outcomes 

What was the EOG proficiency rate of the 3 rd -grade AAA participants in 
2011-12? 

How does the tutoring dosage, i.e., the amount of tutoring received, 
affect the academic outcomes? 

What is the probability of students meeting their growth targets? 

To what extent did students from the AAA spring 2011 academy cohort 
reach proficiency by the end of the 2011-12 school year? 

Cost Analyses 

What is the cost of AAA per student served? Per student reaching 
proficiency? Per student making growth? 
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D&A conducted a short, online survey of the site coordinators to gather their feedback on the 
implementation of AAA at their respective sites. Each of the 25 site coordinators were invited to 
participate in the survey. The response rate was 84%. 

AAA site coordinators from each of the 20 participating schools provided student rosters to 
D&A. Based on the AAA rosters, there were a total of 1,185 students initially enrolled in the 
spring 2012 academy. The participants’ attendance data included in these rosters are 
summarized in this report. Demographic and academic achievement data obtained from end-of- 
year (EOY) student rosters for the 2011-12 school year were available for 1,167 participants 
only. Among these students, academic performance data before (2010-11) and after (201 1-12) 
participation in AAA were available for 818 students in grades 4-8, including 489 who received 
tutoring in reading and 329 participating in mathematics tutoring. Students in grade 3 are 
excluded from all academic analyses because they do not have EOG proficiency or growth data 
for 2010-11; however, their overall 201 1-12 proficiency rate is reported. Figure 1 shows the 
flowchart of the analytical samples. 

Matched Student Comparison Group 

Student outcomes are expected to improve over the course of a school year, so using a matched 
comparison group is especially important for this study. Therefore, to facilitate a more robust 
examination of the academic outcomes of AAA participants, matched comparison groups of 
similar students who did not participate in AAA were selected for the 489 students who received 
tutoring in reading and the 329 students who participated in mathematics tutoring. Although 
these students did not receive tutoring from AAA, they may have received other reading or 
mathematics interventions or programs. 

In reading, a one-to one matched sample was randomly drawn from all students in grades 4-8 in 
2011-12 who had reading EOG achievement levels for 2010-11 and 2011-12 and a reading 
academic change score for 2011-12. Among these eligible students, students were matched 
based their reading EOG achievement level for 2010-11 and several 2011-12 indicators including 
grade level, race/ethnicity, limited English proficiency (LEP) status, economic disadvantage 
indicator, and receipt of special education services (SWD). The one-to-one matched 
mathematics sample was similarly created using mathematics EOG achievement levels in place 
of reading. 

Students who comprised the matched comparison group for reading were not eligible for 
selection for the matched comparison group for mathematics. Thus, the random selections 
yielded two unique groups (489 matched reading students and 329 matched mathematics 
students) who hold similar attributes as the AAA participants for which academic outcomes are 
compared in this report. 
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Figure 1 

Analytical Sample Flowchart 
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Implementation Results 


In 201 1-12, a total of 20 schools participated in AAA. Among the participating schools were 15 
elementary schools and five middle schools, including River Oaks Alternative Middle School. 
Thirteen of the schools also participated in AAA during the 2010-11 school year. Nine of the 10 
elementary schools (excluding Jeffreys Grove Elementary School) were Title I schools in 201 1- 
12 school year One goal set for the spring 2012 academy was to serve between 1,200 and 1,300 
students. According to student rosters provided by site coordinators, 1,185 students participated 
in AAA. The majority of these students (60.7%) were assigned to reading tutoring. Five schools 
offered reading tutoring only and three schools tutored students in mathematics only. Wilburn 
Elementary School had the largest number of participating students (125) and River Oaks had the 
fewest (17). 


Table 3 

AAA Participants by School and Subject, 2011-12 


School 

Total 

# of Students 

Reading 

Mathematics 

All Elementary (n=10) 

938 

561 

377 

Barwell Road Elementary Year Round** 

60 

37 

23 

Brentwood Magnet Elementary** 

72 

72 

0 

Bugg Magnet Elementary * 

73 

63 

10 

Conn Magnet Elementary* 

36 

36 

0 

Creech Road Elementary** 

49 

0 

49 

Fox Road Elementary** 

47 

0 

47 

Hodge Road Elementary Year Round** 

94 

63 

31 

Jeffreys Grove Elementary 

70 

34 

36 

Knightdale Elementary** 

49 

49 

0 

Lincoln Heights Elementary* 

72 

32 

40 

Lynn Road Elementary** 

29 

14 

15 

Poe Magnet Elementary* 

34 

34 

0 

Stough Elementary* 

49 

26 

23 

Walnut Creek Elementary* 

79 

36 

43 

Wilburn Elementary Year Round** 

125 

65 

60 

All Middle (n=5) 

247 

158 

89 

Carroll Middle 

67 

67 

0 

East Garner Magnet Middle 

37 

0 

37 

East Millbrook Magnet Middle 

70 

52 

18 

Moore Square Magnet Middle 

56 

31 

25 

River Oaks Alternative Middle 

17 

8 

9 

Total (N=20) 

1,185 

719 (60.7%) 

466 (39.3%) 


Note: ** indicates schoolwide Title I schools; * indicated targeted assistance Title I schools. 
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Attendance 

Attendance rates are based on the ten weeks of instruction, equaling 20 tutoring sessions each 
lasting 90 minutes. Most schools also offered a review session and a closing celebration, which 
added an extra one or two sessions to the program. Attending 20 sessions (30 hours of 
instruction) was the basis for perfect attendance (100%). Most schools reported attendance rates 
above 100% because they included the extra sessions in their calculations. For example, a 
participation rate of 105% or 100% represents 2 land 22 sessions, respectively 

Nearly all of the schools held between 20 and 22 AAA sessions. The exceptions include Hodge 
Road Elementary School that held 18 sessions only, resulting in a maximum attendance rate of 
90%; and Wilburn Elementary School that had an attendance rate of 1 15% because it held 23 
sessions. 

Table 4 shows the attendance data by school for the 1,185 students. Two trends are evident 
within the data shown; attendance was strong overall yet differs by school level. 

• There is a wide range in the number of AAA sessions that students attended. For example, at 
Barwell Road Elementary School, students attended between 5 and 22 sessions in the spring 
2012 academy yielding a participation rate between 25% and 1 10%. Despite the wide 
participation range, mode data indicate the most students had relatively high attendance. 

• Elementary students had better AAA attendance than middle school students. On average, 
AAA elementary students attended 18 sessions and had an average participation rate of 91%. 
It was most common for elementary students to attend 21 sessions, yielding a 105% 
participation rate. Middle school students averaged 14 sessions and a 71% participation rate, 
although students most frequently attended 17 sessions, resulting in a participation rate of 
85%. 
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Table 4 

AAA Participant Attendance by School, 2011-12 



Total 

Days Attended 

Percent of Days Attended 

School 

Number 
of Students 

Mean 

Mode 

Standard 

Deviation 

Range 

Mean 

Mode 

Standard 

Deviation 

Range 

All Elementary 

938 

18 

21 

3 

5-23 

91% 

105% 

17% 

25 - 115% 

Barwell Road 

60 

16 

19 

5 

5-22 

81% 

95% 

25% 

25 - 110% 

Brentwood 

72 

19 

20 

3 

7-21 

93% 

100% 

15% 

35 - 105% 

Bugg 

73 

20 

21 

2 

5-22 

100% 

105% 

12% 

25 - 110% 

Conn 

36 

18 

21 

4 

5-22 

91% 

105% 

22% 

25 - 110% 

Creech Road 

49 

19 

21 

3 

10-22 

97% 

105% 

15% 

50 - 110% 

Fox Road 

47 

19 

19 

2 

12-21 

95% 

95% 

9% 

60 - 105% 

Hodge Road 

94 

16 

17 

1 

10-18 

82% 

85% 

7% 

50 - 90% 

Jeffreys Grove 

70 

20 

22 

4 

5-22 

98% 

110% 

20% 

25 - 110% 

Knightdale 

49 

19 

21 

3 

6-21 

94% 

105% 

15% 

30 - 105% 

Lincoln Heights 

72 

20 

21 

2 

13-22 

101% 

105% 

8% 

65 - 110% 

Lynn Road 

29 

18 

21 

3 

8-21 

90% 

105% 

17% 

40 - 105% 

Poe 

34 

19 

21 

3 

6-21 

94% 

105% 

17% 

30 - 105% 

Stough 

49 

18 

21 

5 

6-22 

89% 

105% 

25% 

30 - 110% 

Walnut Creek 

79 

18 

18 

1 

15-20 

89% 

90% 

7% 

75 - 100% 

Wilburn 

125 

17 

18 

3 

5-23 

84% 

90% 

17% 

25 - 115% 

All Middle 

247 

14 

17 

4 

2-21 

71% 

85% 

21% 

10 - 105% 

Carroll 

67 

12 

12 

4 

2-21 

62% 

60% 

21% 

10 - 105% 

East Garner 

37 

14 

19 

5 

4-21 

71% 

95% 

23% 

20 - 105% 

East Millbrook 

70 

14 

17 

4 

6-20 

71% 

85% 

18% 

30 - 100% 

Moore Square 

56 

16 

18 

4 

5-21 

79% 

90% 

19% 

25 - 105% 

River Oaks 

17 

16 

17 

4 

8-21 

80% 

85% 

20% 

40 - 105% 

Total 

1,185 

17 

21 

4 

2-23 

87% 

105% 

20% 

10 - 115% 


Note: Attendance data are calculated based on the maximum number of sessions offered at the school and a set 


100% attendance, representing 20 sessions attended. 


17 




Academic Achievement Academy 


D&A Report No. 13.03 


Table 5 reports attendance data for all AAA students and supports the aforementioned finding 
that the AAA tutoring sessions were well attended by student participants. 

• Most students (82%) participated in AAA at least 75% of the time, which amounts to 
attending a minimum of 15 sessions. 

• About one third (35%) of students had perfect attendance. 

• Only a small percentage of students (7%) participated in AAA less than half the time. 

Table 5 

Total AAA Participant Attendance, 2011-12 


Days Attended 

Percent Participation 

# of Students 

% of Students 

Total 

2 to 9 
10 to 14 

Less than 50% 
50% to 70% 

80 

139 

6.8% 

11.7% 

219 (18.4%) 

15 to 17 

75% to 85% 

263 

22.2% 


18 or 19 

90% to 95% 

289 

24.4% 


20 or more 

100% or greater 

414 

34.9% 

966 (81.6%) 


Total 

1,185 

100% 



Note: Shaded cells indicate participation rates at 75% and above. 
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Students Served 

EOY demographic and academic achievement data for the 2011-12 school year were available 
for 1,167 of the AAA students. These data show that the majority of students served in the 
spring 2012 academy were first year participants, whereas a small percentage had also received 
services in the spring 2011 academy. 


Table 6 

Spring 2012 AAA Participants by Year of Participation 


Years of AAA Participation 

Total 

One year (Spring 2012) 

1,003 (85.9%) 

Two years (Spring 2011 and 2012) 

164 (14.1%) 

1,167 


Table 7 presents the demographic characteristics of the students served by AAA. The percentage 
of students within each subgroup are compared to those found in WCPSS overall. Data indicate 
that high percentages of minority and economically disadvantaged student populations 
participated in the spring 2012 academy. 

Race/Ethnicity: The majority (59%) of the AAA students were Black/African American 
students, at a percentage more than twice that of the district; and almost one fourth (24%) of the 
AAA students were Hispanic/Latino, about nine percentage points above the district. 

Gender: The male to female ratio for AAA was the exact inverse of the district. Just over half of 
AAA participants were female (52%). 

Special Programs: Slightly less than three fourths (72%) of AAA students were considered 
economically disadvantaged. AAA served a small percentage (14%) of LEP students yet almost 
twice that of the district. Only a few academically gifted students were enrolled in AAA. 


19 


Academic Achievement Academy 


D&A Report No. 13.03 


Table 7 

AAA Participants' Demographic Characteristics, 2011-12 




AAA 

WCPSS 



(Grades 3-8) 

(Grades K-12) 



# 

% 

% 


American Indian or Alaska Native 

3 

0.3% 

0.4% 


Asian or Pacific Islander 

15 

1.3% 

6.3% 


Black or African American 

691 

59.2% 

24.7% 

Race/Ethnicity 

Hispanic or Latino 

279 

23.9% 

15.0% 


Multi-racial (two or more races) 

39 

3.3% 

4.3 


White 

140 

12.0% 

49.3% 

Gender 

Female 

601 

51.5% 

48.9% 

Male 

566 

48.5% 

51.1% 


Economically Disadvantaged (ED) 

844 

72.3% 

33.3% 

Special 

Limited English Proficiency (LEP) 

163 

14.0% 

7.5% 

Programs 

Special Education (SWD) 

126 

10.8% 

13.4% 


Academically Gifted (AG) 

23 

2.0% 

27.3% 

Total 

1,167 



Data source: WCPSS data were retrieved from http://www.wcpss.net/demographics (Vouk, 2012). 


The spring 2012 AAA academy predominately served elementary students with approximately 
300 students in grades 3, 4, and 5. Considerably fewer middle school students were served 
(about 100 or fewer students per grade level). 


Table 8 

AAA Participants by Grade and Subject, 2011-12 


Grade 

Total 

Reading 

Mathematics 

2 

1 

1 

0 

3 

303 

192 

111 

4 

314 

183 

131 

5 

301 

174 

127 

Grades 2 to 5 

919 (78.7%) 

550 

369 

6 

106 

75 

31 

7 

67 

43 

24 

8 

75 

40 

35 

Grades 6 to 8 

248 (21.3%) 

158 

90 

Total 

1,167 

708 

459 


Data source: End-of-Year (EOY) student rosters from 2011-12. 

Note: One 2 nd -grade student was served, based on EOY grade-level data; 

however, the AAA site coordinator reported him as a 3 ld - grade student. 
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As previously discussed, performing at a EOG relative achievement level between 2.0 and 3.3 
was one of the eligibility criteria for participation in AAA. To determine the extent to which 
participants met this selection criterion, the 2010-11 EOG performance data of students in grades 
4 to 8 in 201 1-12 were examined. Of these students, 77.3% had EOG relative achievement 
levels in either reading or mathematics that fell within the established range. Overall, students 
who received tutoring in mathematics were more likely to have met this eligibility criterion than 
students who were reading tutoring recipients, although the results vary by grade level as shown 
in Table 9. 


Table 9 

AAA Participants' EOG Relative Achievement Levels, 2010-11 
Prior to Participation in the AAA Spring 2012 Academy 


2011-12 
Grade Level 

Total 
Served 
by AAA 

Tested 
Grade Level 

Range 

% between 
2.0 and 3.3 

# missing 
data 

# with 
data 

1 

Reading 

4 

183 

3 

1.40 to 3.75 

65.6% 

7 

176 

5 

174 

4 

1.43 to 4.06 

82.8% 

6 

168 

6 

75 

5 

1.87 to 3.83 

77.3% 

2 

73 

7 

43 

6 

1.54 to 3.73 

76.7% 

1 

42 

8 

40 

7 

1.46 to 4.44 

77.5% 

2 

38 

Total 

515 


1.40 to 4.44 

74.8% 

18 

497 

Mathematics 

4 

131 

3 

1.53 to 3.85 

84.7% 

2 

129 

5 

127 

4 

1.44 to 4.67 

78.0% 

9 

118 

6 

31 

5 

1.81 to 4.56 

77.4% 

2 

29 

7 

24 

6 

1.94 to 3.58 

91.7% 

0 

24 

8 

35 

7 

1.67 to 4.17 

74.3% 

2 

33 

Total 

348 


1.44 to 4.67 

81.0% 

15 

333 


Note: Based on the students with End-of-Year data for 201 1012, there were 863 students in grades 4-8 who 

received tutoring in either reading or mathematics. Of these students, 830 had relative achievement level 


data for 2010-11. 
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Characteristics of AAA Participants and Matched Comparison Groups 

Among the participants of the spring 2012 AAA academy, 818 were in grades 4-8 and had 
achievement level data for 2010-11 (prior to AAA) and 201 1-12 (after AAA). The academic 
outcomes of these 818 students, including 489 who received tutoring in reading and 329 
participating in mathematics tutoring, will be compared in this report to the outcomes of similar 
non-participating students. The academic achievement levels and demographic characteristics of 
AAA participants and the matched comparison groups are presented in Tables 10 through 11. 

Table 10 shows that the AAA groups who received reading or mathematics tutoring and their 
comparison group were performing similarly prior to the spring 2012 academy based on their 
EOG achievement levels in 2010-1 1. Each comparison group also mirrored the grade level 
distribution of the AAA reading and mathematics groups, with the exception of the breakdown at 
Level III. There is a greater percentage of AAA students than comparison students at low Level 
ITT in both reading and mathematics. The average reading and mathematics EOG relative 
achievement levels between the AAA and matched groups were examined and were similar 
across both subjects. The racial/ethnic and special program compositions of the matched groups 
are also comparable to the AAA participant groups as shown in Table 11. 

Table 10 

AAA Participants and Matched Comparison Group 
by Subject and EOG Achievement Levels, 2010-11 


Achievement 

Level 

Reading 

Mathematics 

AAA 

Comparison 

AAA 

Comparison 

n 

% 

n 

% 

n 

% 

n 

% 

Level 1 

76 

15.5% 

77 

15.8% 

21 

6.4% 

21 

6.4% 

Level II 

278 

56.8% 

276 

56.4% 

193 

58.7% 

193 

58.7% 

Level III 

132 

27.0% 

133 

27.2% 

109 

33.1% 

109 

33.1% 

3.00 to 3.29 

104 

21.3% 

46 

9.4% 

88 

26.7% 

30 

9.1% 

3.33 to 3.92 

28 

5.7% 

87 

17.8% 

21 

6.4% 

79 

24.0% 

Level IV 

3 

0.6% 

3 

0.6% 

6 

1.8% 

6 

1.8% 

Total 

489 

100% 

489 

100% 

329 

100% 

329 

100% 
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Table 11 

AAA Participants and Matched Comparison Group 
By Subject and Demographic Characteristics, 2011-12 




Read 

ling 



Mathematics 


Race/Ethnicity 

AAA 

Comparison 

AAA 

Comparison 

n 

% 

n 

% 

n 

% 

n 

% 

American Indian or Alaska Native 

2 

0.4% 

2 

0.4% 

1 

0.3% 

1 

0.3% 

Asian or Pacific Islander 

7 

1.4% 

7 

1.4% 

2 

0.6% 

2 

0.6% 

Black or African American 

297 

60.7% 

297 

60.7% 

206 

62.6% 

206 

62.6% 

Hispanic or Latino 

107 

21.9% 

107 

21.9% 

73 

22.2% 

73 

22.2% 

Multi-racial (two or more races) 

15 

3.1% 

15 

3.1% 

10 

3.0% 

10 

3.0% 

White 

61 

12.5% 

61 

12.5% 

37 

11.3% 

37 

11.3% 

Special Programs 









Economically Disadvantaged (ED) 

338 

69.1% 

338 

69.1% 

246 

74.8% 

246 

74.8% 

Limited English Proficiency (LEP) 

49 

10.0% 

49 

10.0% 

47 

14.3% 

46 

14.0% 

Special Education (SWD) 

46 

9.4% 

47 

9.6% 

38 

11.6% 

36 

10.9% 

Academically Gifted (AG) 

15 

3.1% 

16 

3.3% 

4 

1.2% 

4 

1.2% 

Gender 









Female 

252 

51.5% 

245 

50.1% 

169 

51.4% 

166 

50.5% 

Male 

237 

48.5% 

244 

49.9% 

160 

48.6% 

163 

49.5% 

Total 

489 

100% 

489 

100% 

329 

100% 

329 

100% 


Note: Matched comparison groups were not selected based on academically gifted status or gender although the percentages 
within each indicator are representative of those found among AAA participants. 
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Site Coordinator Survey Results on Implementation 

According to a survey of site coordinators conducted by D&A, the AAA site coordinators of the 
spring 2012 academy held a variety of school positions as shown in Table 12. About one third 
(33%) of the site coordinators were classroom teachers. 

Table 12 

Site Coordinator School Positions 



# 

% 

Classroom Teacher 

7 

33% 

Other (Title 1 Teachers, Social Worker, 
Intervention Behavioral Specialist) 

4 

19% 

Instructional Resource Teacher 

3 

14% 

Literacy Coach 

2 

10% 

Specialist Teacher 

2 

10% 

Special Programs Teacher 

2 

10% 

Guidance Counselor 

1 

5% 

Total 

21 

100% 


Site coordinators were asked to report on the data sources that they used to group students into 
tutor groups. Respondents could select as many data sources as were applicable. Site 
coordinators were most likely to rely on teacher recommendations to assign students to tutor 
groups followed closely by EOG assessment results. Typically, site coordinators relied on 
teacher recommendations to place third grade students and EOG assessment data to create 
student groups for fourth through eighth grade students. 


Table 13 

Data Sources Used to Create Tutor Groups 



# 

% 

Teacher recommendations 

16 

76% 

End of Grade assessments 

15 

71% 

Benchmark assessments 

12 

57% 

Grades 

10 

48% 

Other (AIMSweb data and EVAAS data) 

8 

38% 

Report Cards 

6 

29% 

Profile Cards 

2 

10% 

Case 21 assessments 

2 

10% 
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Site coordinators were asked to indicate their level of agreement to a variety of statements as 
referenced in Table 14. A summary of their results by topic is provided below. 

Engagement and Attendance: According to site coordinators, tutors and students seemed to be 
engaged in the instruction. Each of the respondents agreed that students were engaged in the 
instruction and that tutors were using the instructional materials. Site coordinators reported 
slightly less favorable results for attendance with 67% agreeing that students were rarely absent 
from tutoring sessions. Most (81%) agreed that tutors were present to lead academy sessions. 

Satisfaction: Respondents appear to be satisfied with AAA. Nearly all respondents (95%) 
would recommend AAA to other schools and the vast majority (81%) would consider being a 
site coordinator again. 

Support: According to the survey responses, site coordinators felt supported in their efforts. 
Nearly all (90%) thought that Central Office staff provided useful data and offered support. 
Whereas 85% of respondents believe that their school’s administration provided the support 
needed to coordinate AAA, this percentage is lower than the survey results from the spring 2011 
academy in which 100% of site coordinator respondents agreed with this statement. 

Materials and Training: The survey results show dramatic improvements in the site 
coordinators’ views of the materials and training used in the spring 2012 academy compared to 
survey results from the spring 2011 academy. Nearly all (90%) site coordinators who completed 
the survey in 201 1-12 thought that the instructional materials met the needs of the students 
(compared to only 38% in 2010-11); however, the materials didn’t always meet the needs of the 
tutors. Slightly more than three fourths (77%) of site coordinators stated that the 2011-12 launch 
training session prepared them to assist their tutors in using the materials, whereas only slightly 
more than half (53%) of site coordinators felt the same in 2010-11. 
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Table 14 

Level of Agreement with Various Aspects of AAA Implementation 



Strongly 

Agree 

SA+A 

Disagree 

Strongly 


Agree 

(A) 


(D) 

Disagree 


(SA) 




(SD) 

Engagement and Attendance 






Students appeared to be engaged in the 
instruction. 

48% 

52% 

100% 

0% 

0% 

Tutors seemed to be using the instructional 
materials. 

62% 

38% 

100% 

0% 

0% 

Tutors were consistently present to lead 
tutoring groups. 

24% 

57% 

81% 

14% 

5% 

Students were rarely absent from tutoring 
sessions. 

5% 

62% 

67% 

24% 

10% 

Satisfaction 






1 would recommend the AAA program to 
other schools. 

57% 

38% 

95% 

5% 

0% 

1 would consider being a AAA site 
coordinator during the 2012-13 school year. 

48% 

33% 

81% 

19% 

0% 

Support 






The student rosters provided by Data and 
Accountability were useful for selecting AAA 

43% 

47% 

90% 

10% 

0% 

participants. 

Central Office staff was supportive of my 
efforts to coordinate AAA at my school. 

33% 

57% 

90% 

10% 

0% 

My school's administration provided the 
support 1 needed to coordinate AAA in my 
school. 

52% 

33% 

85% 

14% 

0% 

Materials and Training 






The selected instructional materials met the 
needs of the students. 

10% 

80% 

90% 

10% 

0% 

The selected instructional materials met the 
needs of the tutors. 

The launch training session effectively 

10% 

67% 

77% 

24% 

0% 

prepared me to assist the tutors at my site in 
using the instructional materials. 

10% 

67% 

77% 

24% 

0% 
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The survey concluded with two qualitative questions. Site coordinators were invited to share 
their thoughts on what they considered to be the most effective aspect of AAA and to provide 
their input on ways to make the program more effective. Three themes emerged from the 
generated input from site coordinators as well as three common suggestions for improving AAA. 
As shown in Table 15, the materials selected for the spring 2012 academy were viewed as both a 
most effective aspect and an area in need of improvement. 

Table 15 

Qualitative Input on Program Effectiveness and Areas of Improvement 

Most Effective Aspects Areas of Improvement 

Tutor Quality: Site coordinators were quick to Substitutes: Site coordinators were responsible 
praise the tutors for their hard work and for substituting for absent tutors. They believe 

consistency as well as the quality of instruction that this expectation is too great given the other 

they provided to their students. duties they must perform. Site coordinators 

would like there to be a list of readily available 
tutor substitutes in the event that they are needed 
when a tutor is ill or must attend to an emergency. 

Materials: Many site coordinators mentioned that 
improvements need to be made to the materials. 
Suggestions included using materials that offer pre 
and post assessments as well as interim 
benchmarks, using materials that were more 
scripted, and using materials that were more 
aligned with our district standards and EOG 
assessments and required less "tweeking". 

Small Group Size: The low tutor to student ratio Academy Schedule: Several site coordinators 
which yielded small tutoring groups was seen by wished that the academy had started earlier in the 
site coordinators to be very effective. school year and met once a week for 20 weeks 

instead of twice a week for 10 weeks. 


Effective Materials: Many site coordinators 
mentioned the high quality materials and 
effective curriculum that were used to provide 
reading and mathematics instruction. 
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Student Academic Outcomes 


The examination of academic performance based on EOG achievement levels before and after 
participation in AAA is conducted on 4 th -8 th grade-students, including 489 students who received 
tutoring in reading and their comparison group and 329 students who received mathematics 
tutoring and their comparison group. In addition, the percentages of AAA participants and 
comparison group students who met their reading and mathematics targets for 2011-12 are also 
investigated. 

Reading Achievement 

Table 16 presents the change within each group’s reading EOG achievement levels between 
2010-11 and 2011-12 and compares the degree of that change within and between groups. In 
2010-1 1, a higher percentage of AAA students than comparison students fell within the low 
Level III range. Overall, AAA and comparison students were somewhat more likely to score at 
grade level in 201 1-12. The change in reading EOG achievement levels was more positive for 
students in the comparison group than for AAA participants. 

• Several statistically significant differences in where students fell along the range of EOG 
achievement levels were found. The percentage of comparison students who scored at Level 
II dropped 16.9 percentage points between 2010-11 and 201 1-12, whereas the percentage of 
Level III and IV students rose by 14.3 and 4.3 percentage points. Thus, a larger proportion 
of these students were proficient in reading in 201 1-12 than the previous year. The positive 
change at Level III occurred primarily at low Level HI (3.0 to 3.29). 

• A similar trend of smaller yet statistically significant differences was apparent among the 
AAA participants. There was an 11.9 percentage point decline in students who scored at 
Level II. The percentage of Level III students increased by 9.8 percentage points and almost 
all of this improvement occurred at high Level III (3.3 and above). 

• Considering 2011-12 data only, the percentage of AAA participants who received a Level II 
on the reading EOG in 201 1-12 was statistically higher than the comparison group (45% and 
39.5%). Conversely, the percentage of comparison students who scored within Level IV was 
statistically higher than the AAA students (4.9% and 1.2%). 
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Table 16 

AAA Participants and Matched Comparison Group 
Difference in Reading EOG Achievement Levels, 2010-11 and 2011-12 (N=489) 



2010-11 

2011-12 

Difference 

n 

% 

n 

% 

AAA Participants 

Level 1 

76 

15.5% 

83 

17.0% 

1.5% 

Level II 

278 

56.9% 

220 

45.0% 

-11.9%* 

Level III 

132 

27.0% 

180 

36.8% 

9.8%* 

3.00 to 3.29 

104 

21.3% 

109 

22.3% 

1.0% 

3.33 to 3.92 

28 

5.7% 

71 

14.5% 

8.8%* 

Level IV 

3 

0.6% 

6 

1.2% 

0.6% 

Comparison Group 

Level 1 

77 

15.8% 

69 

14.1% 

-1.7% 

Level II 

276 

56.4% 

193 

39.5% 

-16.9%* 

Level III 

133 

27.2% 

203 

41.5% 

14.3%* 

3.00 to 3.29 

46 

9.4% 

111 

22.7% 

13.3%* 

3.33 to 3.92 

87 

17.8% 

92 

18.8% 

1.0% 

Level IV 

3 

0.6% 

24 

4.9% 

4.3%* 


Note: * indicates a statistically significant difference across years and within groups based 

on a z statistic test for proportions. Bold indicates a statistically higher proportion of 
students within that Level when compared between groups. 


The results presented in Table 16 help to substantiate the finding that although AAA participants 
may have improved their reading proficiency, greater gains occurred among the students who did 
not participate in AAA. Table 17 follows students’ transitions within achievement levels based 
on their reading EOG performance in 2010-11 and 201 1-12. The bold cells on the diagonal 
indicate the percentage of students who scored within the same level. Of particular interest are 
the italicized cells above the diagonal which show the percentage of students who scored within 
a higher level in 201 1-12. Again, reading results are more positive for the comparison students. 

• The largest percentages of students within each group are located on the diagonal, especially 
at Level III where the majority of students remained since 2010-11. 

• The cells above the diagonal show a respectable amount of improvement between 2010-11 
and 2011-12 among both groups, especially the comparison students. 

• The AAA program specifically targeted students at a Level II in 2010-11 and 32.7% of those 
students scored proficient on the reading EOG in 201 1-12. Comparatively, a greater 
percentage of students who did not receive reading tutoring from AAA made this transition 
(40.2%). 


29 





Academic Achievement Academy 


D&A Report No. 13.03 


Table 17 

AAA Participants and Matched Comparison Group 
Transition Matrix for Reading EOG Achievement Levels, 2010-11 and 2011-12 


2010-11 

2011-12 

Level 1 

Level II 

Level III 

Level IV 

Total % and n 

AAA Participants 

Level 1 

47.4% 

40.8% 

11.8% 

0% 

15.5% (76) 

Level II 

14.4% 

52.9% 

32.4% 

0.3% 

56.9% (278) 

Level III 

5.3% 

31.1% 

59.8% 

3.8% 

27.0% (132) 

Level IV 

0% 

33.3% 

66.7% 

0% 

0.6% (3) 

Total % and n 

17.0% (83) 

45.0% (220) 

36.8% (180) 

1.2% (6) 

100% (489) 

Comparison Group 

Level 1 

41.6% 

44.1 % 

14.3% 

0% 

15.8% (77) 

Level II 

12.3% 

47.5% 

39.1% 

1.1% 

56.4% (276) 

Level III 

2.3% 

21.0% 

62.4% 

14.3% 

27.2% (133) 

Level IV 

0% 

0% 

33.3% 

66.7% 

0.6% (3) 

Total % and n 

14.1% (69) 

39.5% (193) 

41.5% (203) 

4.9% (24) 

100% (489) 


Note: Bold indicates same achievement level in 2010-1 1 and 2011-12. Italics indicates a higher 
achievement level in 201 1-12. 
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Mathematics Achievement 

Results from the same analysis of AAA students who received mathematics tutoring and their 
matched comparison group are presented in Tables 18 and 19. In 2010-11, a higher percentage 
of AAA students than comparison students fell within the low Level El range. Overall, AAA 
and comparison students were more likely to score at grade level in 201 1-12. Unlike the trend in 
reading, the change in mathematics EOG achievement levels was more positive for students who 
received AAA tutoring in mathematics than those who did not. 

• There were statistically significant differences in the percentages of students within both 
groups who scored within Levels II and III between 2010-11 and 201 1-12, although those 
differences are more positive for AAA participants. It is these students who were primarily 
targeted for the AAA intervention. 

• Among the AAA group, the percentage of Level III students increased by 19.8 percentage 
points. Improvements among high Level El students (3.3 and above) accounted for this 
impact. There was a 15.8 percentage point increase for the comparison group. The positive 
change at Level III occurred primarily at low Level III (3.0 to 3.29). 

• The comparison group had a statistically higher percentage of students at Level IV in 201 1- 
12 than the AAA group. 


Table 18 

AAA Participants and Matched Comparison Group 
Difference in Mathematics EOG Achievement Levels, 2010-11 and 2011-12 (N=329) 



2010-11 

2011-12 

Difference 


n 

% 

n 

% 

AAA Participants 

Level 1 

21 

6.4% 

30 

9.1% 

2.7% 

Level II 

193 

58.7% 

114 

34.7% 

-25.0%* 

Level III 

109 

33.1% 

174 

52.9% 

19.8%* 

3.00 to 3.29 

88 

26.7% 

96 

29.2% 

2.5% 

3.33 to 3.92 

21 

6.4% 

78 

23.7% 

17.3%* 

Level IV 

6 

1.8% 

11 

3.3% 

1.5% 

Comparison Group 

Level 1 

21 

6.4% 

27 

8.2% 

1.8% 

Level II 

193 

58.7% 

120 

36.5% 

-22.2%* 

Level III 

109 

33.1% 

161 

48.9% 

15.8%* 

3.00 to 3.29 

30 

9.1% 

69 

21.0% 

11.9%* 

3.33 to 3.92 

79 

24.0% 

92 

27.9% 

3.9% 

Level IV 

6 

1.8% 

21 

6.4% 

4.6%* 


Note: * indicates a statistically significant difference across years and within groups 

based on a z statistic test for proportions. Bold indicates a statistically higher 
proportion of students within that Level when compared between groups. 
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The results presented in Table 19 support the finding that AAA participants had greater gains in 
their mathematics proficiency than non-participants at Levels II and III. At the targeted levels, 
among AAA students who fell within a Level II on the mathematics EOG in 2010-11, 44.5% of 
those students were proficient in 201 1-12 compared to 39.9% of comparison students. AAA 
students at Level III prior to receiving tutoring were also slightly more likely to remain at that 
level, although the comparison students were more likely to advance to Level IV. 


Table 19 

AAA Participants and Matched Comparison Group 
Transition Matrix for Mathematics EOG Achievement Level, 2010-11 and 2011-12 


2010-11 

2011-12 

Level 1 

Level II 

Level III 

Level IV 

Total % and n 

AAA Participants 

Level 1 

28.6% 

52.4% 

19.0% 

0% 

6.4% (21) 

Level II 

11.9% 

43.5% 

43.5% 

1.0% 

58.7% (193) 

Level III 

0.9% 

17.4% 

76.2% 

5.5% 

33.1% (109) 

Level IV 

0% 

0% 

50.0% 

50.0% 

1.8% (6) 

Total % and n 

9.1% (30) 

34.7% (114) 

52.9% (174) 

3.3% (11) 

100% (329) 

Comparison Group 

Level 1 

14.3% 

71.4% 

14.3% 

0% 

6.4% (21) 

Level II 

12.4% 

47.7% 

39.9% 

0% 

58.7% (193) 

Level III 

0% 

11.9% 

71.6% 

16.5% 

33.1% (109) 

Level IV 

0% 

0% 

50.0% 

50.0% 

1.8% (6) 

Total % and n 

8.2% (27) 

36.5% (120) 

48.9% (161) 

6.4% (21) 

100% (329) 


Note: 


Bold indicates same achievement level in 2010-11 and 2011-12. Italics indicates a higher achievement 
level in 2011-12. Includes students with both 2010-11 and 2011-12 data. 
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Academic Growth in Reading and Mathematics 

Table 20 shows the percentages of students who met their reading growth targets for 201 1-12. 
The results are presented for AAA participants who received reading tutoring and their 
comparison group and are disaggregated by reading EOG achievement level. Little difference 
was found between the percentages of the students who received reading tutoring and met 
growth compared to those students who did not participate in AAA and met growth. None of the 
differences are statistically significant. 

• As previously discussed, AAA primarily targeted students whose achievement levels were 
within the range of 2.0 to 3.3. Less than half of the Level II students in each group met their 
reading growth targets. A greater proportion of students with achievement levels between 
3.0 and 3.29 reached growth (three fourths of students within each group), although higher 
range Level III (3.33 to 3.92) students were even more successful. 

• The percentages of AAA students and their matched group who met reading growth were 
higher than all elementary and middle school students districtwide at every achievement 
level. 

Table 20 

AAA Participants and Matched Comparison Group 
Who Met Reading Growth Targets by Reading EOG Achievement Level, 2011-12 


2011-12 

Achievement 

Level 

AAA Students 
Met Growth 

Comparison Students 
Met Growth 

Difference in % 

# 

% 

# 

% 

Level 1 

16 

19.3% 

11 

15.9% 

3.4% 

Level II 

99 

45.0% 

84 

43.5% 

1.5% 

Level III 

150 

83.3% 

166 

81.8% 

1.5% 

3.00 to 3.29 

83 

76.1% 

83 

75.5% 

0.6% 

3.33 to 3.92 

67 

94.4% 

83 

89.2% 

5.2% 

Level IV 

6 

100% 

23 

95.8% 

4.2% 

Total 

271 

55.4% 

284 

58.1% 

-2.7% 


Note: No differences are statistically significant based on a z statistic test for proportions. 


Table 21 

Reading Growth Targets Met Districtwide, 2011-12 


2011-12 

Achievement Level 

WCPSS Elementary Schools 
Grades 3-5 

WCPSS Middle Schools 
Grades 6-8 

Levels 1 & II 

55% 

55% 

Level III 

62% 

59% 

Level IV 

57% 

54% 

Total 

59% 

57% 
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There are higher percentages of all AAA students who reached growth in mathematics (67.8%) 
compared to reading (55.4%) Overall, the proportions of AAA students by EOG achievement 
level who received mathematics tutoring and who met their mathematics growth targets are 
similar to the comparison students. The only statistically significant difference is found among 
Level III students. 

• It appears that the performance of the higher Level III students (3.31 to 3.92) contributes to 
most of this positive change. The percentage of AAA participants with achievement levels 
between 3.31 and 3.92 on the mathematics EOG in 201 1-12 who made growth was 
statistically higher than the comparison group (96.6% and 85.7%). There was also a 
difference of almost 10 percentage points between the percentage of AAA students and their 
matched counterparts at lower Level III who made growth, yet this difference was not 
significant. 

• The percentages of AAA students and their matched group who were proficient in 
mathematics and met their growth targets were higher than all elementary and middle school 
students districtwide. 

Table 22 

AAA Participants and Matched Comparison Group 
Who Met Mathematics Growth Targets by Mathematics EOG Achievement Level, 2011-12 


2011-12 

Achievement 

Level 

AAA Students 
Met Growth 

Comparison 
Students 
Met Growth 

Difference in % 

# 

% 

# 

% 

Level 1 

2 

6.7% 

1 

3.7% 

3.0% 

Level II 

52 

45.6% 

61 

50.8% 

-5.2% 

Level III 

159 

91.4% 

132 

82.0% 

9.4%* 

3.00 to 3.27 

74 

86.0% 

48 

76.2% 

9.8% 

3.31 to 3.92 

85 

96.6% 

84 

85.7% 

10.9%* 

Level IV 

10 

90.9% 

20 

95.2% 

-4.3% 

Total 

223 

67.8% 

214 

65.1% 

2.7% 


Note: * indicates a statistically significant difference based on a z statistic test for proportions. 


Table 23 

Mathematics Growth Targets Met Districtwide, 2011-12 


2011-12 

Achievement Level 

WCPSS Elementary Schools 
Grades 3-5 

WCPSS Middle Schools 
Grades 6-8 

Levels 1 & II 

62% 

59% 

Level III 

71% 

60% 

Level IV 

73% 

61% 

Total 

71% 

60% 
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Table 24 presents the percentage of students who met their growth targets by grade level and 
subject. Again, there is little evidence of a positive impact of AAA on students’ academic 
growth. Overall, the difference in proportions reaching growth is minimal between groups and 
only two instances of significance are found. 

The comparison group had a statistically higher percentage of 6 th -grade students meeting their 
reading growth targets than the 6 th -grade students who received reading tutoring (56.9% 
compared to 41.7%). Shifting attention to mathematics, slightly more than three fourths of AAA 
students in grade 5 met growth in 201 1-12, resulting in a statistically significant difference of 
13.1 percentage points more than 5 th -grade comparison group students. 

Table 24 

AAA Participants and Matched Comparison Group 
Who Met Reading or Mathematics Growth Targets by Grade Level, 2011-12 


2011-12 
Grade Level 

AAA Students 
Met Growth 

Comparison Students 
Met Growth 

Difference in % 

# 

% 

# 

% 


Reading 

4 

89 

51 . 2 % 

96 

55 . 2 % 

- 4 . 0 % 

5 

103 

62 . 4 % 

98 

59 . 4 % 

3 . 0 % 

6 

30 

41 . 7 % 

41 

56 . 9 % 

- 15 . 2 %* 

7 

25 

61 . 0 % 

22 

53 . 7 % 

7 . 3 % 

8 

24 

64 . 9 % 

27 

73 . 0 % 

- 8 . 1 % 

Total 

271 

55.4% 

284 

58.1% 

-2.7% 

Mathematics 

4 

88 

68 . 2 % 

89 

69 . 0 % 

- 0 . 8 % 

5 

90 

78 . 3 % 

75 

65 . 2 % 

13 . 1 %* 

6 

13 

46 . 4 % 

16 

57 . 1 % 

- 10 . 7 % 

7 

9 

37 . 5 % 

12 

50 . 0 % 

- 12 . 5 % 

8 

23 

69 . 7 % 

22 

66 . 7 % 

3 . 0 % 

Total 

223 

67.8% 

214 

65.1% 

2.7% 


Note: * indicates a statistically significant difference based on a z statistic test for proportions. 
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Academic growth results by subgroup are displayed in Tables 25 and 26 for AAA participants 
and their comparison groups. 

• In most cases, the majority of students in each group met their reading growth targets. The 
AAA students had slightly higher percentages of students meeting growth than comparison 
students within three subgroup categories; Hispanic/Latino, LEP, and special education, 
although these difference were not statistically significant. 

• Compared to reading outcomes, greater proportions of the student subgroups within each 
analytic group met their mathematics growth targets. Among almost all subgroups, it was 
slightly more common for AAA students who received mathematics tutoring to have met 
growth than the comparison students. None of the differences were statistically significant. 


Table 25 

AAA Participants and Matched Comparison Group 
Who Met Reading Growth Targets by Demographic Characteristics, 2011-12 



AAA Students 
Met Growth 

Comparison Students 
Met Growth 

Difference in 

% 

Race/Ethnicity 

# 

% 

# 

% 

American Indian or Alaska Native 

ns 

ns 

ns 

ns 

ns 

Asian or Pacific Islander 

ns 

ns 

ns 

ns 

ns 

Black or African American 

166 

55.9% 

168 

56.6% 

-0.7% 

Hispanic or Latino 

60 

56.1% 

57 

53.3% 

2.8% 

Multi-racial (two or more races) 

8 

53.3% 

12 

80.0% 

-26.7% 

White 

33 

54.1% 

40 

65.6% 

-11.5% 

Special Programs 






Free or Reduced-Price Lunch 

189 

55.9% 

190 

56.2% 

-0.3% 

Limited English Proficiency 

31 

63.3% 

26 

53.1% 

10.2% 

Special Education 

28 

60.0% 

24 

53.3% 

6.7% 

Academically Gifted 

7 

46.7% 

10 

71.4% 

-24.7% 

Gender 






Female 

146 

57.9% 

149 

60.8% 

-2.9% 

Male 

125 

52.7% 

135 

55.3% 

-2.6% 


Note: No differences are statistically significant based on a z statistic test for proportions. 

Growth data for subgroups with 5 or fewer students are not shown (ns). Students may be counted in more than 
one special program. 
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Table 26 

AAA Participants and Matched Comparison Group 
Who Met Mathematics Growth Targets by Demographic Characteristics, 2011-12 



AAA Students 
Met Growth 

Comparison Students 
Met Growth 

Difference in 

% 

Race/Ethnicity 

# 

% 

# 

% 

American Indian or Alaska Native 

ns 

ns 

ns 

ns 

ns 

Asian or Pacific Islander 

ns 

ns 

ns 

ns 

ns 

Black or African American 

136 

66.0% 

135 

65.5% 

0.5% 

Hispanic or Latino 

52 

71.2% 

48 

65.8% 

5.4% 

Multi-racial (two or more races) 

7 

70.0% 

7 

70.0% 

0.0% 

White 

26 

70.3% 

21 

56.8% 

13.5% 

Special Programs 






Free or Reduced-Price Lunch 

165 

69.9% 

155 

63.0% 

6.9% 

Limited English Proficiency 

33 

70.2% 

29 

63.0% 

7.2% 

Special Education 

23 

60.5% 

21 

58.3% 

2.2% 

Academically Gifted 

Gender 

ns 

ns 

ns 

ns 

ns 

Female 

118 

69.8% 

109 

65.7% 

4.1% 

Male 

105 

65.6% 

105 

64.4% 

1.2% 


Note: No differences are statistically significant based on a z statistic test for proportions. 

Growth data for subgroups with fewer than 5 students are not shown (ns). Students may be counted in more than 
one special program. 
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School Level Academic Outcomes 


Academic Outcomes of AAA Participants by School 

AAA participants’ reading and mathematics EOG proficiency and growth data are presented by 
school in Tables 27 and 28. These data allow comparisons to be made across schools that 
implemented AAA in the spring of 2012. Fewer than half of the schools experienced gains in the 
percentage of students who scored proficient after receiving tutoring. Yet there appears to be a 
positive relationship between proficiency and growth, as the schools that showed the greatest 
improvement in AAA student proficiency tend to have the highest percentages of AAA students 
meeting growth. 

• Among the 20 participating schools, 17 offered tutoring in reading. Based on EOG results 
for 2010-11 and 2011-12, there were three schools that had significant improvements in the 
percentage of their AAA participants who were proficient in reading following the tutoring 
they received, as shown in Table 27. These schools are Conn and Walnut Creek Elementary 
and Carroll Middle. Barwell Road, Jeffrey’s Grove, and Lynn Road Elementary and River 
Oaks Middle also had notable, but non-statistically significant gains in reading proficiency 
among their AAA students. 

• At 12 of the 17 schools, more than half of the AAA students met their reading growth targets 
for 2011-12. Five of the schools had at least 65% of their AAA students making reading 
growth, and these schools also had the greatest rise in student proficiency (Barwell Road, 
Conn, Jeffrey’s Grove, and Walnut Creek Elementary and River Oaks Middle). 

• Fifteen of the schools offered tutoring in mathematics during the spring 2012 academy. Two 
of the schools, Walnut Creek 1 2 and Wilburn Elementary Schools, had significant increases in 
the percentage of students who scored proficient on the mathematics EOG after they received 
tutoring. Many of the same schools that showed improvement in reading also had notable 
but non-statistically significant gains in AAA students’ mathematics proficiency (Barwell 
Road , Bugg, and Lynn Road Elementary and River Oaks Middle). 

• More than half of AAA students met their mathematics growth targets for 201 1-12 at 12 of 
the 15 schools. At least 75% of the AAA students made growth in mathematics at 7 of the 
schools that in most cases also had notable improvements in proficiency (Barwell Road, 
Bugg, Jeffrey’s Grove, Lynn Road, Stough, Walnut Creek, and Wilburn Elementary 
Schools). 


1 Walnut Creek had multiple reading and mathematics interventions in place for the 2011-12 school year 
and it is unknown whether students might have received those supports in addition to AAA. 

2 Barwell Road, Brentwood, Creech Road, and Wilburn were Renaissance Schools in 2011-12 and were 
provided extra resources specifically related to technology, professional development, additional staff, a 
signing bonus for most staff, and a performance bonus plan rewarding growth in student performance, 
with the goal of increasing student performance at these low performing schools (Lenard et al., 2012). 
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Table 27 
AAA Schools 

Reading EOG Proficiency and Growth Data (N=20) 


School 

Number of 
Students 

Proficiency 

2010-11 

Proficiency 

2011-12 

Difference 

in 

Proficiency 

% Meeting 
Growth 
Targets 
2011-12 

Barwell Road Elementary Year 
Round 

24 

16.7% 

37.5% 

20.8% 

70.8% 

Brentwood Magnet Elementary 

42 

35.7% 

40.5% 

4.8% 

52.4% 

Bugg Magnet Elementary 

35 

45.7% 

45.7% 

0.0% 

57.1% 

Conn Magnet Elementary 

19 

42.1% 

79.0% 

36.9%* 

68.4% 

Creech Road Elementary 

nd 

nd 

nd 

nd 

nd 

Fox Road Elementary 

nd 

nd 

nd 

nd 

nd 

Hodge Road Elementary Year 
Round 

38 

23.7% 

36.8% 

13.1% 

63.2% 

Jeffreys Grove Elementary 

21 

28.6% 

52.4% 

23.8% 

71.4% 

Knightdale Elementary 

31 

22.6% 

41.9% 

19.3% 

45.2% 

Lincoln Heights Elementary 

17 

47.1% 

47.1% 

0.0% 

52.9% 

Lynn Road Elementary 

9 

0% 

22.2% 

22.2% 

33.3% 

Poe Magnet Elementary 

18 

44.4% 

33.3% 

-11.1% 

50.0% 

Stough Elementary 

11 

36.4% 

36.4% 

0.0% 

36.4% 

Walnut Creek Elementary 

27 

22.2% 

48.2% 

26.0%* 

66.7% 

Wilburn Elementary Year Round 

48 

8.3% 

18.8% 

10.5% 

52.1% 

Carroll Middle 

65 

16.9% 

33.9% 

17.0%* 

49.2% 

East Garner Magnet Middle 

nd 

nd 

nd 

nd 

nd 

East Millbrook Magnet Middle 

47 

34.0% 

23.4% 

-10.6% 

51.1% 

Moore Square Magnet Middle 

30 

36.7% 

40.0% 

3.3% 

53.3% 

River Oaks Alternative Middle 

7 

28.6% 

57.1% 

28.5% 

85.7% 

Total 

489 






Note: Bold schools are first year AAA schools. Italics indicate Renaissance Schools. Creech Road and Fox Road 
Elementary and East Garner Middle Schools did not offer reading tutoring and therefore have no data (nd). 
Walnut Creek opened in 2011-12, so the students it served in AAA that year attended other elementary 
schools in 2010-11. * indicates a statistically significant difference based on a z statistic test for proportions. 


39 



Academic Achievement Academy 


D&A Report No. 13.03 


Table 28 
AAA Schools 

Mathematics EOG Proficiency and Growth Data (N=20) 


School 

Number of 
Students 

Proficiency 

2010-11 

Proficiency 

2011-12 

Difference 

in 

Proficiency 

% Meeting 
Growth 
Targets 
2011-12 

Barwell Road Elementary Year 
Round 

9 

22.2% 

55.6% 

33.4% 

77.8% 

Brentwood Magnet Elementary 

nd 

nd 

nd 

nd 

nd 

Bugg Magnet Elementary 

5 

20.0% 

60.0% 

40.0% 

100% 

Conn Magnet Elementary 

nd 

nd 

nd 

nd 

nd 

Creech Road Elementary 

31 

51.6% 

71.0% 

19.4% 

71.0% 

Fox Road Elementary 

25 

32.0% 

48.0% 

16.0% 

60.0% 

Hodge Road Elementary Year 
Round 

30 

43.3% 

50.0% 

6.7% 

50.0% 

Jeffreys Grove Elementary 

22 

68.2% 

81.8% 

13.6% 

90.9% 

Knightdale Elementary 

nd 

nd 

nd 

nd 

nd 

Lincoln Heights Elementary 

28 

35.7% 

53.6% 

17.9% 

53.6% 

Lynn Road Elementary 

9 

33.3% 

55.6% 

22.3% 

77.8% 

Poe Magnet Elementary 

nd 

nd 

nd 

nd 

nd 

Stough Elementary 

16 

50.0% 

75.0% 

25.0% 

75.0% 

Walnut Creek Elementary 

33 

27.3% 

75.8% 

48.5%* 

87.9% 

Wilburn Elementary Year Round 

37 

27.0% 

62.2% 

35.2%* 

83.8% 

Carroll Middle 

nd 

nd 

nd 

nd 

nd 

East Garner Magnet Middle 

32 

15.6% 

34.4% 

18.8% 

65.6% 

East Millbrook Magnet Middle 

18 

33.3% 

27.8% 

-5.5% 

50.0% 

Moore Square Magnet Middle 

25 

36.0% 

48.0% 

12.0% 

36.0% 

River Oaks Alternative Middle 

9 

0% 

22.2% 

22.2% 

66.7% 

Total 

329 






Note: Bold schools are first year AAA schools. Italics indicate Renaissance Schools. Brentwood, Conn, 

Knightdale, and Poe Elementary and Carroll Middle Schools did not offer mathematics tutoring and therefore 
have no data (nd). Walnut Creek opened in 2011-12, so the students it served in AAA that year attended 
other elementary schools in 2010-1 1. * indicates a statistically significant difference based on a z statistic 
test for proportions. 
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AAA School Performance Composites 

Table 29 shows the ABCs performance composites for all schools participating in AAA. Eight 
of the 20 schools (40%) had at least a 70% performance composite by 201 1-12. More than half 
of the schools that held a spring 2012 academy also participated in the spring of 201 1. These 13 
second- year schools had performance composites below 70% in 2009-10 and did not reach 70% 
after their first year of implementation in 2010-1 1. Thus, they were invited to offer AAA at their 
schools in 2011-12. 

• After a second year of participating in AAA, four of the 13 schools, Barwell Road and Poe 
Elementary and East Garner and Moore Square Middle, had performance composites slightly 
over 70%, representing a statistically significant improvement compared to 2009-10 and 
2010 - 11 . 

• Compared to 2009-10, the year prior to AAA implementation, Brentwood, Lynn Road, and 
Wilburn Elementary also experienced a significant increase over time, although their 
performance composites did not reach 70%. Conversely, Carroll Middle School experienced 
a significant decline. 

• By 201 1-12, four of the first year schools had restored their performance composite to at 
least 70%, which was a significant increase for Bugg, Conn, and Jeffreys Grove Elementary 
Schools. River Oaks Middle’s performance composite represented a significant drop 
between 2010-11 and 2011-12. 
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Table 29 

AAA Schools by Year of Participation 
ABCs Performance Composites, 2009-10, 2010-11, 2011-12 (N=20) 



2009-10 ABCs 

2010-11 ABCs 

2011-12 ABCs 

Difference 

Difference 


Performance 

Performance 

Performance 

2009-10 to 

2010-11 to 

School 

Composite 

Composite 

Composite 

2011-12 

2011-12 


Second Year Schools (Spring 2011 and Spring 2012) n=13 


Borwell Road Elementary Year 
Round 

53.0% 

64.4% 

74.1% 

21.1* 

9.7* 

Brentwood Magnet Elementary 

56.1% 

61.0% 

64.1% 

8.0* 

3.1 

Creech Road Elementary 

53.9% 

57.1% 

58.1% 

4.2 

1.0 

Fox Road Elementary 

63.0% 

68.9% 

66.5% 

3.5 

-2.4 

Hodge Road Elementary Year 
Round 

64.8% 

62.5% 

64.1% 

-0.7 

1.6 

Lincoln Heights Elementary 

69.0% 

67.7% 

68.0% 

-1.0 

0.3 

Lynn Road Elementary 

61.2% 

68.3% 

66.1% 

4.9* 

-2.2 

Poe Magnet Elementary 

60.5% 

68.3% 

74.0% 

13.6* 

5.7* 

Wilburn Elementary Year Round 

53.9% 

58.7% 

66.4% 

12.5* 

7.7* 

Carroll Middle 

69.3% 

64.1% 

63.6% 

-5.7* 

-0.5 

East Garner Middle 

67.7% 

69.6% 

71.8% 

4.1* 

2.2* 

East Millbrook Magnet Middle 

66.0% 

64.8% 

65.3% 

-0.7 

0.5 

Moore Square Magnet Middle 

62.2% 

69.8% 

73.0% 

10.8* 

3.2* 


First Year Schools (Spring 2012) n=7 


Bugg Magnet Elementary 

75.7% 

63.1% 

70.6% 


7.5* 

Conn Magnet Elementary 

71.5% 

69.6% 

75.9% 


6.3* 

Jeffreys Grove Elementary 

72.7% 

66.8% 

83.3% 


16.5* 

Knightdale Elementary 

70.4% 

69.3% 

69.7% 


0.4 

Stough Elementary 

70.9% 

68.3% 

72.5% 


4.2 

Walnut Creek Elementary 

NA 

NA 

60.3% 


NA 

River Oaks Alternative Middle 

53.1% 

67.3% 

57.8% 


-9.5* 


Note: Bold performance composites for 2011-12 indicate those above 70.0%. Italics indicate Renaissance Schools. 
* indicates a statistically significant difference based on a z statistic test for proportions. Positive and 
significant improvements are in italics. Difference between 2009-10 and 2011-12 are not shown for first 
year schools because 2010-11 was the year prior to AAA implementation. 
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Additional Analyses of Outcomes 


Several questions arose during this evaluation that led to additional analyses of academic 

outcomes. 

• Third grade students who participated in AAA during the 2011-12 school year were not 
included in the previous student and school-level analyses because they lack prior EOG 
proficiency and growth target data, due to the nature of the state assessments,. Their 
performance on the EOG after they received AAA tutoring is still of interest and is examined 
here. 

• The second set of analyses investigates whether students who receive higher doses of 
tutoring have more positive academic outcomes than students with lower doses. First, the 
tutoring dosage that students received is examined based on the number of tutoring sessions 
that students attended during the spring 2012 academy. Next, the academic outcomes of 
students who received AAA tutoring in the same subject in two doses (during the spring 
2011 and 2012 academies) are compared to students who received one dose of tutoring in the 
spring 2012 academy. 

• Next is an analysis of the probability that students would meet their growth targets by the end 
on the 2011-12 school year. Logistic regression models are applied to the sample of students 
who received reading or mathematics tutoring in the spring 2012 academy and their 
comparisons groups. 

• The final analysis explores the extent to which students from the AAA spring 2011 academy 
cohort reached proficiency by the end of the 2011-12 school year. In this case, three-year 
reading and mathematics proficiency trend data for the spring 201 1 academy cohort are 
examined to assess whether this group of students had reached proficiency. 


Grade 3 EOG Proficiency 

During the AAA 2012 academy, 192 students in grade 3 received tutoring in reading and 111 
students were tutored in mathematics. Soon after the 1 1 week academy, the students took the 
EOG assessments. These students were likely selected for AAA participation based on other 
assessment and achievement data sources which may have indicated below grade-level 
performance. 

EOG data for 2011-12 indicate that 3 rd -grade students performed better in mathematics than 
reading. Among the students who were tutored in reading, about one-third comprised Levels I, 
II, and III, respectively. As such, 35% of the students were reading proficient by the end of the 
year. A larger percentage of mathematics tutor recipients were proficient (56%) as shown in 
Table 30. 
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Table 30 

Grade 3 AAA Participants 
EOG Achievement Level, 2011-12 


2011-12 

Achievement 

Level 

Reading 

Mathematics 

# 

% 

# 

% 

Level 1 

58 

30.2% 

6 

5.4% 

Level II 

67 

34.9% 

43 

38.7% 

Level III 

62 

32.3% 

57 

51.4% 

Level IV 

5 

2.6% 

5 

4.5% 

Total 

192 

100% 

111 

100% 


Tutoring Dosage 

AAA Attendance 

One question that surfaced during this evaluation was whether the tutoring dosage based on 
students’ attendance at the tutoring sessions would make any difference in the reading or 
mathematics outcomes that they were expected to gain by the end of the spring 2012 academy. 
To investigate this question, students in grades 4-8 who received reading tutoring were separated 
into three groups based on their attendance. The groups’ proficiency rates were then compared. 
The same procedure was used for those students tutored in mathematics. Students with 
attendance rates lower than 50% (2 to 9 sessions) are labeled the “low attendance group,” 
students with rates between 50-70% (10 to 14 sessions) comprise the “medium attendance 
group,” and students who attended the academy at least 75% of the time (15 or more sessions) 
are called the “high attendance group.” Group assignments are based on attendance data 
presented in Table 5. It showed the most students fell into the high attendance group. 
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It was expected that after the spring 2012 academy, students with high attendance would have 

higher reading and mathematics proficiency rates than the middle and low attendance groups. 

The results, as shown in Figures 2 through 4, indicate that high attendance did not make a 

difference in EOG proficiency or growth outcomes for 201 1-12. 

• Students in the low attending group who received tutoring in reading from AAA started out 
with slightly lower reading proficiency levels than higher attending students prior to the 
academy (2010-11). Students who were tutored in mathematics had similar pre-AAA 
mathematics proficiency levels across the attendance groups. By 2011-12, the percentages of 
low attending students who were proficient in reading or mathematics were roughly 
equivalent to those found among high attending students, although low attending students 
had the largest gains in proficiency. 

• The one-year reading and mathematics proficiency gains made by both low and high 
attending students are statistically significant. 

• The percentages of students meeting their reading or mathematics growth targets in 201 1-12 
are also similar among low and high attendance groups, as shown in Figure 4. 

• Students who attended 10-14 sessions (the medium attendance group) had the lowest 
academic outcomes and the smallest gains in proficiency. 


Figure 2 

Reading Proficiency Before and After AAA by Students' Tutoring Attendance (n=489) 



Interpretation Example: Before receiving reading tutoring from AAA (in 2010-11), 20.5% of the low 

attending students were proficient in reading. This percentage increased to 41.0% 
in 2011-12, after their participation in AAA. 
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Figure 3 

Mathematics Proficiency Before and After AAA by Students' Tutoring Attendance (n=329) 
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Figure 4 

Reading and Mathematics Growth for 2011-12 by Students' AAA Attendance 
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AAA Tutoring: Two Years Compared to One Year 

In 2010-11, AAA served 1,357 students, and 1,185 students were served in 2011-12. A small 
percentage (14.1%) of the students served by AAA in 2010-11 was also served in 201 1-12. Of 
the students who participated in AAA for two years, 104 were served in the same subject, as 
shown in Table 31. Restricting the analysis to students in grades 4-8, the academic outcomes of 
students who received tutoring in both the spring 2011 and 2012 academies are compared to the 
students who received tutoring in the spring 2012 academy only. The purpose was to evaluate 
whether a double dose of the AAA intervention positively impacts proficiency and growth more 
than a single dose. 


Table 31 

Spring 2012 AAA Participants by Years of Reading or Mathematics Tutoring Received 


Year of AAA Tutoring Received in the 
Same Subject 

Reading 

Mathematics 

Total 

One year (Spring 2012) 

645 

418 

1,063 

Two years (Spring 2011 and 2012) 

63 

41 

104 


708 

459 

1,167 


EOG proficiency and growth data shown in Figures 5 through 7 suggest that overall students 
who participated in both AAA academies and received tutoring in the same subject did not have 
considerably better end of year outcomes than students who were first year participants in 201 1- 
12. A second dosage of reading tutoring does appear to have a small positive impact on low 
performing students, when the gains between the first and second academy are examined. 

• In 2010-1 1, the reading proficiency of students was lower among those who received another 
year of literacy tutoring than students who were tutored one year only, which may have 
deemed them eligible candidates for participation in the second academy. This group’s 
reading proficiency did double after a second dose of tutoring, which is a statistically 
significant improvement, although their overall rate was still lower than students who 
received reading tutoring in the spring 2012 academy only. The reading gains made by the 
one-year participants are statistically significant as well. 

• Almost identical, statistically significant gains in mathematics proficiency were also found 
for students who participated either one or two years. 

• The percentages of students meeting their reading or mathematics growth targets are also 
similar among students regardless of their years of participation. 
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Figure 5 

Reading Prof iciency for Students Tutored in 
Reading by Years of Participation 
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Figure 6 

Mathematics Proficiency for Students Tutored in 
Mathematics by Years of Participation 
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Figure 7 

Reading and Mathematics Growth for 2011-12 by Years of Participation 
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Probability of Meeting Growth Targets 

The next analysis investigated the probability of students meeting their reading and mathematics 
growth targets. Logistic regression models were used to see if student characteristics such as 
grade level, race/ethnicity, participation in special programs (ED, LEP, SWD), and 2011-12 
proficiency levels impacted whether or not students in grades 4-8 would meet their 2011-12 
growth targets. One model was applied to students who received AAA reading tutoring and their 
comparison group and the other model included students who received AAA mathematics 
tutoring and their comparison group. 

• Being proficient on the 2011-12 reading EOG exam significantly increased the odds of 
meeting growth in reading that same year, regardless of AAA participation. 

• AAA reading tutoring did appear to have a small yet statistically significant positive impact 
on LEP students. For instance, these students were twice as likely to have met their reading 
growth targets as other students who received reading tutoring and were not identified as 
LEP. 

• The odds of meeting mathematics growth in 201 1-12 significantly increased when students 
were proficient on the mathematics EOG exam in the spring of 2012. This impact occurred 
for both students who received AAA tutoring and those who did not, yet the extent of the 
effect was greater for AAA students. 
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2010-11 AAA Cohort Outcomes 

One of the outcomes articulated in the AAA 2010-11 evaluation is that the cohort of students 
who participated in AAA during the 2010-11 school year would reach proficiency by 2011-12 
(Baenen & Lougee, 2011). Analysis was conducted on the students in this cohort who had EOG 
data for 2009-10 (prior to AAA), 2010-11 (AAA), and 2011-12 (after AAA). The three-year 
trend data on the reading and mathematics proficiency of this group of students indicates that 
most were not proficient in 201 1-12. These students’ proficiency appears to be greatest during 
AAA implementation and lowest before and after. 

• In 2010-11, 42.9% of the students tutored in reading during the spring 201 1 academy were 
proficient. This represents a statistically significant increase of almost 1 1 percentage points 
compared to the proportion of students who were proficient prior to AAA in 2009-10 
(32.1%). The percentage of students who took the reading EOG exam in 201 1-12 and 
reached proficiency declined to 35.7%. 

• The mathematics proficiency rate of the students who received mathematics tutoring during 
the 201 1 spring academy remained relatively stable between 2009-10 (58%) and 2010-11 
(59.7%). Among the students who took the mathematics EOG exam in 2011-12, 49.5% were 
proficient, representing a one-year statistically significant decline of about 10 percentage 
points. 


Table 32 

Three-Year Proficiency Trend for 
AAA 2010-11 Participants 



2009-10 

2010-11 (AAA) 

2011-12 

Difference 

Difference 


(Prior to AAA) 

(Grades 3-8) 

(After AAA) 



Subject 

n 

% 

n 

% 

n 

% 

2009-10 to 
2010-11 

2010-11 to 
2011-12 

Reading 

169 

32.1% 

298 

42.9% 

222 

35.7% 

10.8%* 

-7.2% 

Mathematics 

269 

58.0% 

364 

59.7% 

263 

49.5% 

1.7% 

-10.2%* 


Note: Students in grade 3 in 2010-11 are not included in 2009-10 data; Students in grade 8 in 2010-11 are not 
included in 201 1-12 data. 
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Cost of AAA 


The AAA budget for 2011-12 was allotted based on the expenditures of the spring 2011 
academy. Budgeted costs for staff benefits, instructional supplies and materials, and 
transportation for students increased from amounts expended in 2010-11, whereas reductions 
were made in salary compensation for tutors and staff development allotments. The overall 
budget for the spring 2012 academy was over half a million dollars ($645,619). Nearly all 
(95.8%) of this budgeted amount was spent ($618,233). Almost $20,000 of the anticipated costs 
for transporting students home after the tutoring sessions was not spent. 


Table 33 

Academic Achievement Academy Budget, 2011-12 



2010-11 

Expenditures 

2011-12 

Budget 

2011-12 

Expenditures 

Budget 

Balance 

Tutorial Pay 

$335,476.75 

$326,784.56 

$324,807.31 

$1,977.25 

Staff Benefits 

$72,672.46 

$78,541.91 

$73,655.22 

$4,886.69 

Staff Development 

$35,280.00 

$13,272.00 

$13,272.00 

$0.00 

Supplies and Materials 

$103,908.99 

$117,874.04 

$117,332.24 

$541.80 

Student Transportation 

$102,241.67 

$109,146.88 

$89,166.15 

$19,980.73 

Total 

$649,579.87 

$645,619.39 

$618,232.92 

$27,386.47 


Table 34 provides the estimated program cost per student participant. By dividing the total 
expenditure amount by the total number of student participants, the cost of AAA is estimated to 
be slightly over $500 for each participant. This figure is greater than the estimate calculated for 
2010-12 by $43 more per student because fewer students were served in 201 1-12 than in 2010- 
11. One third (414) of the students had 100% attendance in the spring 2012 academy. If all 
1,185 students had fully participated in the 30 hours of reading or mathematics offered by AAA, 
the hourly cost per student would have been $17.39. 


Table 34 

Estimated Total Cost per AAA Participant, 2011-12 



2011-12 

2010-11 

Annual 

Difference 

Total Expenditures 

$618,233 

$649,580 

($31,347) 

# of Student Participants 

1,185 

1,357 

($172) 

Total Cost Per Participant 

$521.72 

$478.69 

$43 
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The cost of providing tutoring through AAA during the spring 2012 academy increases 
considerably if the cost is calculated per proficient student or per student who made academic 
growth. Proficiency calculations are presented for comparative purposes. Making academic 
growth was the primary short-term goal of AAA student participants whereas reaching 
proficiency was more long-term. Based on available achievement data for the participating 
students in grades 3-8, the cost per student reaching proficiency is over $1,000 ($1,152). The 
cost per student in grades 4-8 who made academic growth is under $1,000 ($864). The provision 
of reading tutoring is considerably more expensive than mathematics tutoring. 

Table 35 

Estimated Cost per AAA Participant 
Meeting Reading or Mathematics Proficiency, 2011-12 



Grade 3-8 Students 

EOG Proficiency 2011-12 

Subject 

# 

Total cost 

#/% reaching proficiency 

Cost per student 
reaching proficiency 

Reading 

707 

$368,856 

268 (37.9%) 

$1,376.33 

Mathematics 

459 

$239,469 

260 (56.6%) 

$921.04 

Total 

1,166 

$608,326 

528 (45.3%) 

$1,152.13 


Note: The total cost for all students in grades 3-8 with assessment data is calculated by taking the total cost 
per student ($521.72) multiplied by the number of students in grades 3-8 (1,166) who have available 
data. The same logic is applied to the total cost of students receiving reading tutoring and those 
receiving mathematics tutoring. 


Table 36 

Estimated Cost per AAA Participant 
Meeting Reading or Mathematics Growth Targets, 2011-12 



Grade 4-8 Students 

EOG Growth Targets 2011-12 

Subject 

# 

Total cost 

#/% meeting 
growth targets 

Cost per student 
making growth 

Reading 

489 

$255,121 

271 (55.4%) 

$941.41 

Mathematics 

329 

$171,646 

223 (67.8%) 

$769.71 

Total 

818 

$426,767 

494 (60.4%) 

$863.90 


Note: The total cost for all students in grades 4-8 is calculated by taking the total cost per student 

($521.72) multiplied by the number of students in grades 4-8 (818) who have available growth 
target data. The same logic is applied to the total cost of students receiving reading tutoring and 
those receiving mathematics tutoring. 
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